Solid-state imaging device and electronic device

ABSTRACT

A solid-state imaging device ( 10 ) includes an imager ( 11 ) configured to acquire image data, a processing unit ( 14 ) configured to perform a process based on a neural network calculation model for data based on the image data acquired from the imager, and a control unit ( 12 ) configured to switch between a first process mode of performing a first process at a first frame rate and, based on a result of the first process, a second process mode of performing a second process at a second frame rate.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 17/661,857, filed on May 3, 2022, which is acontinuation application of U.S. patent application Ser. No. 17/250,438,filed on Jan. 21, 2021, now U.S. Pat. No. 11,350,046, which is a U.S.National Phase of International Patent Application No. PCT/JP2019/030092filed on Jul. 31, 2019, which claims the benefit of priority fromJapanese Patent Application No. JP 2019-140824 filed in the Japan PatentOffice on Jul. 31, 2019, which claims the benefit of priority fromJapanese Patent Application No. JP 201 8-144172 filed in the JapanPatent Office on Jul. 31, 2018. Each of the above-referencedapplications is hereby incorporated herein by reference in its entirety.

FIELD

The present disclosure relates to a solid-state imaging device and anelectronic device. Specifically, the present disclosure relates todynamic control of frame rate.

BACKGROUND

A device such as a digital camera is equipped with an image sensorhaving a complementary metal oxide semiconductor (CMOS) and a digitalsignal processor (DSP). In the image sensor, the captured image issupplied to the DSP and undergoes a variety of processing in the DSP tobe output to an external device such as an application processor.

CITATION LIST Patent Literature

Patent Literature 1: WO2018/051809

SUMMARY Technical Problem

Unfortunately, in the conventional technique above, typically, simpleimage processing such as noise removal is performed in the DSP in theimage sensor, and complicated processing such as facial recognitionusing image data is performed in the application processor. Sincedesigning the DSP appropriate for an application and installing anapplication for each process impose a heavy burden on developers andusers, it has been desired that more complicated processing is performedin the chip of the image sensor.

The present disclosure then proposes a solid-state imaging device and anelectronic device capable of performing more complicated processing inthe chip of the image sensor.

Solution to Problem

To solve the above-described problem, a solid-state imaging deviceaccording to one aspect of the present disclosure includes an imagerconfigured to acquire image data, a processing unit configured toperform a process based on a neural network calculation model for databased on the image data acquired from the imager, and a control unitconfigured to switch between a first process mode of performing a firstprocess at a first frame rate and, based on a result of the firstprocess, a second process mode of performing a second process at asecond frame rate.

Moreover, according to the present disclosure, a solid-state imagingdevice includes a DSP to run a DNN (Deep Neural Network) in a chip,changes the frame rate dynamically depending on whether the DSP processis performed by DSP, and implements execution of a complicated processin the chip.

Advantageous Effects of Invention

According to the present disclosure, more complicated processing can beperformed in the chip of the image sensor. The effect described here isnot always limitative, and any other effects described in the presentdisclosure may be achieved.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an overall configuration exampleof an imaging device as an electronic device according to a firstembodiment.

FIG. 2A is a diagram illustrating a first process mode according to thefirst embodiment.

FIG. 2B is a diagram illustrating a second process mode according to thefirst embodiment.

FIG. 3 is a flowchart illustrating a process mode selecting processaccording to the first embodiment.

FIG. 4 is a flowchart illustrating the first process mode according tothe first embodiment.

FIG. 5 is a diagram illustrating a modification to the first embodiment.

FIG. 6 is a diagram illustrating a modification to the first embodiment.

FIG. 7 is a diagram illustrating a chip configuration example of animage sensor according to the present embodiment.

FIG. 8 is a diagram illustrating a layout example according to thepresent embodiment.

FIG. 9 is a diagram illustrating a layout example according to thepresent embodiment.

FIG. 10 is a block diagram illustrating an example of the overallconfiguration of a vehicle control system.

FIG. 11 is a diagram illustrating an example of the installationposition of a vehicle exterior information detector and an imager.

FIG. 12 is a diagram illustrating an example of the overallconfiguration of an endoscopic surgery system.

FIG. 13 is a block diagram illustrating an example of the functionalconfiguration of a camera head and a CCU.

DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure will be described in detail belowwith reference to the drawings. In the following embodiments, the sameparts are denoted by the same reference signs and an overlappingdescription will be omitted.

The present disclosure will be described in the order of items below.

1. First Embodiment 2. Modification (1) to First Embodiment 3.Modification (2) to First Embodiment 4. Chip Configuration of ImageSensor 5. Layout Example 6. Other Embodiments 7. Application to MovableBody 8. Application to Endoscopic Surgery System 1. First Embodiment1-1. Configuration of Image Processing System According to FirstEmbodiment

FIG. 1 is a block diagram illustrating an overall configuration exampleof an imaging device as an electronic device according to a firstembodiment. As illustrated in FIG. 1 , an imaging device 1 is connectedto communicate with a cloud server 30. The imaging device 1 and thecloud server 30 are connected either wired or wirelessly to communicatethrough a variety of networks and universal serial bus (USB) cables.

The cloud server 30 is an example of a server device that stores imagedata such as still images and moving images transmitted from the imagingdevice 1. For example, the cloud server 30 can store image data in anyunits, such as by users, by days, or by image capturing locations, andprovide a variety of services such as creating a photobook using imagedata.

The imaging device 1 is an example of an electronic device having animage sensor 10 and an application processor 20. Examples includedigital cameras, digital camcorders, tablet terminals, and smartphones.Although the following embodiments are described with an example ofcapturing an image, the present invention is not limited thereto and,for example, a moving image can also be processed similarly.

The image sensor 10 is, for example, a complementary metal oxidesemiconductor (CMOS) image sensor configured in one chip, receivesincident light, performs photoelectric conversion, and outputs imagedata corresponding to the quantity of received incident light to theapplication processor 20.

The application processor 20 is an example of a processor such as acentral processing unit (CPU) that executes a variety of applications.The application processor 20 performs a variety of processescorresponding to applications, such as a display process of displayingimage data input from the image sensor 10 on a display, a biometricauthentication process using image data, and a transmission process oftransmitting image data to the cloud server 30.

Such an image sensor 10 of the imaging device 1 has a DSP in a chip torun a DNN, thereby implementing execution of a complicated process inthe chip. However, when a DNN is installed in the image sensor 10, areadout process of a captured image and a DSP process of performing acomplicated process are sometimes performed concurrently. If a DSPprocess is performed concurrently with an image readout process, noiseresulting from the DSP process may intrude into the read image data todeteriorate the quality of the image. Then, in the image sensor 10according to the first embodiment, quality deterioration of an image issuppressed by controlling the execution period of the DSP processrelative to the execution period of the image readout process.

1.2 Configuration of Imaging Device According to First Embodiment

As illustrated in FIG. 1 , the imaging device 1 includes the imagesensor 10 that is a solid-state imaging device and the applicationprocessor 20. The image sensor 10 includes an imager 11, a controller(also referred to as control unit) 12, a signal processor 13, a DSP(also referred to as processing unit) 14, a memory 15, and a selector16.

The imager 11 includes, for example, an optical system 104 including azoom lens, a focus lens, and an aperture, and a pixel array 101 having aconfiguration in which unit pixels including light-receiving elementssuch as photodiodes are arranged in a two-dimensional matrix. Externalincident light passes through the optical system 104 to form an image ona light-receiving surface that is an array of light-receiving elementsin the pixel array 101. Each unit pixel in the pixel array 101 convertslight incident on its light-receiving element into electricity toaccumulate charge in accordance with the quantity of incident light sothat the charge can be read out.

The imager 11 includes a converter (analog-to-digital converter,hereinafter referred to as ADC) 17 (for example, see FIGS. 2A and 2B).The ADC 17 converts an analog pixel signal for each unit pixel read fromthe imager 11 to a digital value to generate digital image data andoutputs the generated image data to the signal processor 13. The ADC 17may include a voltage generating circuit that generates a drive voltagefor driving the imager 11 from power supply voltage and the like.

The size of image data output by the imager 11 can be selected from aplurality of sizes, for example, 12 M (3968×2976) pixels and a videographics array (VGA) size (640×480 pixels Z). The image data output bythe imager 11 can be selected from, for example, RGB (red, green, blue)color image or white and black image only having brightness. Theseselections can be made as a kind of shooting mode settings.

The controller 12 controls each part in the image sensor 10, forexample, in accordance with user's operation or an operation mode beingset.

For example, the controller 12 monitors the image readout process ofreading out an image signal from the imager 11 described later, and uponcompletion of the image readout process, notifies the DSP 14 to startthe DSP process. Specifically, the controller 12 counts the internalclock and switches an enable signal to control whether to overlap theDSP process with the readout process of image data from the imager 11(hereinafter simply referred to as image readout process).

For example, the controller 12 outputs a LOW enable signal (that is, OFFsignal) to the DSP 14 while the image readout process from the imager 11is being performed. Upon completion of the image readout process fromthe imager 11, the controller 12 outputs a HIGH enable signal (that is,ON signal) to the DSP 14. Subsequently, upon completion of the DSPprocess by the DSP 14, the controller 12 outputs a LOW enable signal(that is, OFF signal) to the DSP 14.

The signal processor 13 performs a variety of signal processing fordigital image data read from the imager 11 or digital image data readfrom the memory 15 (hereinafter referred to as process target imagedata). For example, when the process target image data is a color image,the signal processor 13 converts the format of this image data to YUVimage data, RGB image data, or the like. The signal processor 13performs, for example, processing such as noise removal and whitebalance adjustment for the process target image data, if necessary. Inaddition, the signal processor 13 performs a variety of signalprocessing (also referred to as pre-processing) for the process targetimage data in order for the DSP 14 to process the image data.

The DSP 14 executes, for example, a computer program stored in thememory 15 to function as a processing unit that performs a variety ofprocessing using a pre-trained model created by machine learning using adeep neural network (DNN). For example, the DSP 14 performs acomputation process based on the pre-trained model stored in the memory15 to perform a process of combining image data with a dictionarycoefficient stored in the memory 15. The result obtained through such acomputation process (computation result) is output to the memory 15and/or the selector 16. The computation result may include image dataobtained by performing a computation process using the pre-trained modeland a variety of information (metadata) obtained from the image data. Amemory controller for controlling access to the memory 15 may beembedded in the DSP 14.

More specifically, when the enable signal output from the controller 12changes from LOW (OFF) to HIGH (ON), the DSP 14 reads out image datafrom the memory 15 and performs the DSP process. That is, the DSP 14performs the DSP process in a period of time in which the enable signalis ON.

The image data to be processed by the DSP 14 may be image data normallyread out from the pixel array 101 or may be image data having a datasize reduced by decimating pixels of the image data normally read out.Alternatively, the image data to be processed may be image data read outin a data size smaller than normal obtained by performing readout fromthe pixel array 101 with pixels decimated. As used herein “normalreadout” may be readout without decimating pixels.

Some computation processes use, for example, a pre-trained learningmodel that is an example of a neural network calculation model. Examplesof the pre-trained learning model include DNNs and support vectormachines that have learned identification of persons and animals,brightness correction, face extraction, and image correction usingtraining data. The DSP process uses, for example, the pre-trainedlearning model to specify a person from a captured image, correctsbrightness or noise of a captured image, or extract a face from acaptured image. The DSP 14 can perform the computation process describedabove by training a learning model by changing the weights of a varietyof parameters in the learning model using training data, preparing aplurality of learning models and changing a learning model to be used inaccordance with a computation process, or acquiring a pre-trainedlearning model from an external device.

The memory 15 stores image data output from the imager 11, image datasubjected to signal processing by the signal processor 13, a computationresult obtained from the DSP 14, and the like, if necessary. The memory15 also stores an algorithm of the pre-trained learning model to beexecuted by the DSP 14 in the form of a computer program and adictionary coefficient.

The memory 15 may store the International Organization forStandardization (ISO) sensitivity, exposure time, frame rate, focus,shooting mode, cutout range, and the like, in addition to image dataoutput from the signal processor 13 and image data subjected to acomputation process output from the DSP 14 (hereinafter referred to asprocessed image data). That is, the memory 15 may store a variety ofimaging information set by the user.

The selector 16, for example, selectively outputs image data output fromthe signal processor 13 or image data or a computation result stored inthe memory 15, in accordance with a select control signal from thecontroller 12. One of the processed image data and the computationresult such as metadata stored in the memory 15 is selected by theuser's setting, for example, and output to the application processor 20.

More specifically, when a first process mode is selected, the selector16 reads the computation result generated by the DSP 14 from the memory15 and outputs the read computation result to the application processor20. On the other hand, when a second process mode is selected, theselector 16 outputs image data input from the signal processor 13 to theapplication processor 20. When the first process mode is selected, theselector 16 may output the computation result output from the DSP 14directly to the application processor 20.

As described above, the image data or the computation result output fromthe selector 16 is input to the application processor 20 that processesdisplay and user interface. The application processor 20 is configured,for example, with a central processing unit (CPU) and executes anoperating system and a variety of application software. This applicationprocessor 20 may be equipped with functions such as a graphicsprocessing unit (GPU) and a baseband processor. The applicationprocessor 20 performs a variety of processes for the input image data orthe computation result as necessary, or performs display to users, ortransmits the input image data or the computation result to an externalcloud server 30 through a predetermined network 40.

For example, a variety of networks such as the Internet, a wired localarea network (LAN) or a wireless LAN, a mobile communication network, orBluetooth (registered trademark) can be applied to the predeterminednetwork 40. The image data or the computation result may be transmittednot only to the cloud server 30 but also to a variety of informationprocessing devices (systems) having a communication function, such as aserver operating on its own, a file server storing a variety of data,and a communication terminal such as a mobile phone.

1.3 Description of Process Modes According to First Embodiment

The process modes according to the first embodiment will now bedescribed. FIG. 2A is a diagram illustrating the first process modeaccording to the first embodiment, and FIG. 2B is a diagram illustratingthe second process mode according to the first embodiment. FIG. 2Aillustrates the first process mode processed at a first frame rate, andFIG. 2B illustrates the second process mode processed at a second framerate that is higher than the first frame rate. For example, the firstprocess mode in FIG. 2A is processed at 1 frame rate (fps), and thesecond process mode in FIG. 2B is processed at 30 frame rate.

As illustrated in FIG. 2A, when the first process mode is selected, thecontroller 12 keeps the enable signal LOW while the image readoutprocess from the imager 11 is being performed and, upon completion ofthe image readout process, changes the enable signal to HIGH. Whendetecting that the enable signal is changed to HIGH, the DSP 14 startsthe DSP process. Subsequently, upon completion of the DSP process, thecontroller 12 changes the enable signal to LOW. As a result, the nextimage readout process is performed in the imager 11.

As illustrated in FIG. 2B, since the enable signal is always kept LOWwhile the second process mode is selected, the image readout processfrom the imager 11 is successively performed in accordance with theframe rate.

For example, the switching of the process modes illustrated in FIGS. 2Aand. 2B can be applied to a security camera. Specifically, while theenable signal is set to HIGH and the first process mode illustrated inFIG. 2A is selected, the controller 12 reads out an image from theimager 11, and the DSP 14 performs human presence detection from theread image. When the DSP 14 detects a person, the controller 12 sets theenable signal to LOW and performs only the process of reading out andoutputting an image from the imager 11. In this way, the image readoutprocess and the human presence detection by the DSP 14 are performeduntil a suspicious person or the like is detected, and upon detection ofa suspicious person, the computation process based on a neural networkcalculation model is eliminated, and the image readout process alone isprioritized. The application processor 20 therefore can implement earlydetection of a suspicious person, can grasp the motion of the detectedsuspicious person, and can monitor the suspicious person. When a personis no longer detected, the first process mode returns.

The switching of the process modes illustrated in FIGS. 2A and 2B can beapplied to a camera that shoots athletes of a variety of sports.Specifically, while the enable signal is set to HIGH and the firstprocess mode illustrated in FIG. 2A is selected, the controller 12 readsout an image of a tennis player from the imager 11, and the DSP 14detects a state of the tennis player holding a racket from the readimage. When the DSP 14 detects this state, the controller 12 sets theenable signal to LOW and performs only the process of reading out andoutputting an image from the imager 11. In this way, the image readoutprocess and the human presence detection by the DSP 14 are performeduntil the tennis player hits a ball with the racket, and upon detectionof the timing when a ball is hit, the image readout process alone isprioritized. The application processor 20 therefore can specify themotion of hitting a ball from the start to the end at an earlier stageand can provide output to users. When a state of finishing swinging theracket ends, the first process mode returns.

As described above, in the second process mode, when a certain detectiontarget such as a person is detected by the DSP process of the DSP 14 inthe first process mode, the DSP process by the DSP 14 is not performedbut the image data readout process by the controller 12 is performed atthe second frame rate that is higher than the first frame rate in thefirst process mode. This eliminates the need for wait time in which thecompletion of the DSP process is waited in the first process mode. Inthe second process mode, therefore, the image readout process can beperformed at the second frame rate that is a frame rate that is higher(for example, twice) than the first frame rate in the first processmode.

1.4 Flow of Process According to First Embodiment

FIG. 3 is a flowchart illustrating a process mode selecting processaccording to the first embodiment. As illustrated in FIG. 3 , thecontroller 12 displays a process mode select screen on a display or thelike and accepts selection of a process mode (S101).

If the first process mode is selected (Yes at S102), the controller 12sets the frame rate to the first frame rate (S103). Subsequently, theimage readout process in the first process mode is performed (S104).

On the other hand, if the second process mode is selected (No at S102),the controller 12 sets the frame rate to the second frame rate that ishigher than the first frame rate (S105). Subsequently, the image readoutprocess in the second process mode is performed (S106).

When the process is terminated, for example, by the operation ofterminating the image capturing process or by power-off (Yes at S107),resetting of the settings is performed. On the other hand, if theprocess continues (No at S107), the process subsequent to S101 isrepeated. That is, the previous process mode is kept until the userchanges the process mode, and when the process mode is changed, theframe rate is changed in accordance with FIG. 3 .

FIG. 4 is a flowchart illustrating the first process mode according tothe first embodiment. The process in FIG. 4 is performed at S104 inFIGS. 2A and 2B.

As illustrated in FIG. 4 , upon start of image capturing in the firstprocess mode, the controller 12 starts the image readout process ofreading out image data from the imager 11 (S201). Upon completion of theimage readout process from the imager 11 (Yes at S202), the controller12 changes the enable signal to ACTIVE (ON) (S203).

Subsequently, the DSP 14 performs the DSP process (S204). Uponcompletion of the DSP process (Yes at S205), the controller 12 changesthe enable signal to INACTIVE (OFF) (S206). Subsequently, if imagecapturing continues (No at S207), the process returns to S201. On theother hand, if, for example, the operation to terminate image capturingis accepted (Yes at S207), the process ends. If an operation to changeprocess modes occurs even during the process in FIG. 4 , the process inFIG. 3 is performed.

1.5 Operation Effects

As described above, the image sensor 10 can allow the DSP 14 to operatein a software manner by allowing the DSP 14 to run a DNN in a chip,thereby performing a complicated process through the DSP process using aDNN in one chip. The image sensor 10 can change the frame ratedynamically depending on whether the DSP process is performed inaccordance with the process mode, achieve higher speed of processing,and improve the user's usability. The image sensor 10 can control thestart of the DSP process with an enable signal and therefore cansuppress whether to perform the readout process of an image from theimager 11 and the DSP process concurrently, thereby suppressingoccurrence of noise and suppressing image quality deterioration of theimage sensor.

2. Modification (1) to First Embodiment

In the foregoing first embodiment, suppression of noise occurrence andnoise reduction are performed by controlling the execution period of theDSP process by the DSP 14 with an enable signal. However, the presentinvention is not limited thereto. When noise is acceptable to somedegree, for example, in image capturing in bright light, the imagereadout process and the DSP process may be performed concurrently. Anexample of performing the image readout process and the DSP processconcurrently is described here.

FIG. 5 is a diagram illustrating a modification to the first embodiment.As illustrated in FIG. 5 , when noise is acceptable to some degree in astate in which the first process mode of performing the image readoutprocess and the DSP process is selected, the image readout process andthe DSP process can be performed concurrently.

For example, when a predetermined condition is satisfied, the controller12 changes a period in which the enable signal is activated.Specifically, as illustrated in FIG. 5 , after the image readout processis started, the controller 12 changes the enable signal to ACTIVE duringexecution of the image readout process to allow the image readoutprocess and the DSP process to be performed concurrently. As used herein“when a predetermined condition is satisfied” may refer to when noise isacceptable to some degree, for example, when the quantity of light equalto or larger than a preset threshold is obtained.

The second process mode illustrated in FIG. 5 can be applied to, forexample, a monitoring camera for keeping an eye on infants and elderlypersons since the image data readout process by the controller 12 can betemporally overlapped with the DSP process by the DSP 14 for data basedon the image data. Specifically, while the first process modeillustrated in FIG. 2A is selected, the controller 12 reads an imagefrom the imager 11, and the DSP 14 performs human presence detectionfrom the read image. When the DSP 14 detects a person, the controller 12sets an ACTIVE period of the enable signal to increase the frame ratecompared with the first process mode. As a result, the DSP 14 can detectthe posture of the person appearing in the image, from an image,concurrently with the image readout by the controller 12. That is, theimage readout process with a small process load and the posturedetection with a large process load are performed concurrently. In thisway, the image readout process and the human presence detection by theDSP 14 are performed until a monitoring target is detected, and upondetection of the monitoring target, more sophisticated detection ofposture of the monitoring target can be performed. The applicationprocessor 20 therefore can achieve both of the early detection of amonitoring target and the detection of motion of the detected monitoringtarget. When a person is no longer detected, the first process modereturns.

In this way, in the second process mode illustrated in FIG. 5 , when acertain detection target is detected by the DSP process by the DSP 14 inthe first process mode, the DSP process by the DSP 14 can be performedat the second frame rate that is the same frame rate as the first framerate in the first process mode. When noise is acceptable to some degree,therefore, the DSP process can be started without waiting for completionof the image readout process. Consequently, even when the calculationvolume of the DSP process is large and a long process time is required,reduction in the frame rate can be avoided, or, if the frame rate isreduced, the amount of reduction can be reduced.

3. Modification (2) to First Embodiment

In the foregoing embodiment, in the first process mode, the imagereadout process and the DSP process are performed in different timeslots. However, the present invention is not limited to thereto. Forexample, when the DSP process is light and the process time is short,the first process mode can include a normal mode at the first frame rate(FIG. 2A) and a speed priority mode at a frame rate that is higher thanthe first frame rate.

FIG. 6 is a diagram illustrating a modification to the first embodiment.As illustrated in FIG. 6 , in the speed priority mode, the image readoutprocess and the DSP process are performed concurrently, and uponcompletion of the DSP process, the next image readout is performedwithout delay. That is, in the second process mode illustrated in FIG. 6, the DSP process is performed at the second frame rate that is higherthan the first frame rate in the first process mode when a certaindetection target is detected by the DSP process in the first processmode.

When the speed priority mode is preset by the user, for example, thecontroller 12 activates the enable signal during execution of the imagereadout process to allow the DSP process to be performed concurrently.Subsequently, upon completion of the DSP process, the controller 12immediately inactivates the enable signal and allows the next imagereadout process to be performed.

The process mode illustrated in FIG. 6 can be applied to imageprocessing for home-use games and Internet games. Specifically, whilethe first process mode illustrated in FIG. 2A is selected, thecontroller 12 reads out a game image reflecting the user's operationfrom the imager 11, and the DSP 14 detects a pose specific to the readgame image. Then, when the DSP 14 detects a specific pose, thecontroller 12 sets the speed priority mode illustrated in FIG. 6 toincrease the frame rate. As a result, the readout of a game image by thecontroller 12 and the pose detection by the DSP 14 can be performedsimultaneously. When a killer move using a gesture is made in a game,the application processor 20 can perform pose detection at high speedand determine a motion necessary for the killer move at high speed. Whena specific motion is no longer detected, the first process mode returns.

In this way, even when the DSP process is performed similarly to thefirst process mode (FIG. 2A), the process can be performed at a framerate similar to that in the second process mode (FIG. 2B), therebyachieving higher speed of the process.

4. Chip Configuration of Image Sensor

An example of the chip configuration of the image sensor 10 illustratedin FIG. 1 will now be described in detail below with reference to thedrawings.

FIG. 7 is a diagram illustrating a chip configuration of the imagesensor according to the present embodiment. As illustrated in FIG. 7 ,the image sensor 10 has a stack structure in which a first substrate(die) 100 shaped like a quadrangular flat plate and a second substrate(die) 120 similarly shaped like a quadrangular flat plate are bondedtogether.

The first substrate 100 and the second substrate may have the same size,for example. The first substrate 100 and the second substrate 120 eachmay be a semiconductor substrate such as a silicon substrate.

In the first substrate 100, in the configuration of the image sensor 10illustrated in FIG. 1 , the pixel array 101 of the imager 11 isarranged. A part or the whole of the optical system 104 may be providedon a chip in the first substrate 100.

In the second substrate 120, in the configuration of the image sensor 10illustrated in FIG. 1 , the ADC 17, the controller 12, the signalprocessor 13, the DSP 14, the memory 15, and the selector 16 arearranged. A not-illustrated interface circuit, driver circuit, and thelike may be arranged in the second substrate 120.

The first substrate 100 and the second substrate 120 may be bondedtogether by chip-on-chip (CoC) technology in which the first substrate100 and the second substrate 120 are individually diced into chips, andthese diced first substrate 100 and second substrate 120 are bondedtogether, or by chip-on-wafer (CoW) technology in which one of the firstsubstrate 100 and the second substrate 120 (for example, the firstsubstrate 100) is diced into a chip, and the diced first substrate 100is bonded to the second substrate 120 before dicing (that is, in a waferstate), or by wafer-on-wafer (WoW) technology in which the firstsubstrate 100 and the second substrate 120 both in a wafer state arebonded together.

For example, plasma joining can be used as a joining process between thefirst substrate 100 and the second substrate 120. However, the presentinvention is not limited thereto and a variety of joining processes maybe used.

5. Layout Example

FIG. 8 and FIG. 9 are diagrams for explaining a layout example accordingto the present embodiment. FIG. 8 illustrates a layout example of thefirst substrate 100, and FIG. 9 illustrates a layout example of thesecond substrate 120.

5-1. Layout Example of First Substrate

As illustrated in FIG. 8 , in the first substrate 100, in theconfiguration of the image sensor 10 illustrated in FIG. 1 , the pixelarray 101 of the imager 11 is arranged. When a part or the whole of theoptical system 104 is mounted on the first substrate 100, it is providedat a position corresponding to the pixel array 101.

The pixel array 101 is arranged off-center to one side L101 among foursides L101 to L104 of the first substrate 100. In other words, the pixelarray 101 is arranged such that its center O101 is more proximate to theside L101 than the center O100 of the first substrate 100. When thesurface having the pixel array 101 in the first substrate 100 isrectangular, the side L101 may be, for example, a shorter side. However,the present invention is not limited thereto, and the pixel array 101may be arranged off-center to a longer side.

In a region proximate to the side L101 among four sides of the pixelarray 101, in other words, a region between the side L101 and the pixelarray 101, a TSV array 102 is provided, in which a plurality of throughsilicon vias (hereinafter referred to as TSVs) passing through the firstsubstrate 100 are arranged as wiring for electrically connecting eachunit pixel 101 a in the pixel array 101 to the ADC 17 arranged in thesecond substrate 120. In this way, the TSV array 102 is provided inproximity to the side L101 proximate to the pixel array 101 to ensure aspace for each part such as the ADC 17 in the second substrate 120.

The TSV array 102 may also be provided in a region proximate to one sideL104 (or may be the side L103) of two sides L103 and L104 intersectingthe side L101, in other words, in a region between the side L104 (or theside L103) and the pixel array 101.

A pad array 103 having a plurality of pads arranged linearly is providedon each of the sides L102 and L103 on which the pixel array 101 is notarranged off-center, among four sides L101 to L104 of the firstsubstrate 100. The pads included in the pad array 103 include, forexample, a pad (also referred to as power supply pin) receiving powersupply voltage for analog circuits such as the pixel array 101 and theADC 17, a pad (also referred to as power supply pin) receiving powersupply voltage for digital circuits such as the signal processor 13, theDSP 14, the memory 15, the selector 16, and the controller 12, a pad(also referred to as signal pin) for interfaces such as a mobileindustry processor interface (MIPI) and a serial peripheral interface(SPI), and a pad (also referred to as signal pin) for input/output ofclock and data. Each pad is electrically connected to, for example, anexternal power supply circuit or an interface circuit through a wire. Itis preferable that each pad array 103 and the TSV array 102 aresufficiently spaced apart to such a degree that influences of reflectionof signals from the wire connected to each pad in the pad array 103 canbe ignored.

5-2. Layout Example of Second Substrate

On the other hand, as illustrated in FIG. 9 , in the second substrate120, in the configuration of the image sensor 10 illustrated in FIG. 1 ,the ADC 17, the controller 12, the signal processor 13, the DSP 14, andthe memory 15 are arranged. In the first layout example, the memory 15is divided into two regions: a memory 15A and a memory 15B. Similarly,the ADC 17 is divided into two regions: an ADC 17A and adigital-to-analog converter (DAC) 17B. The DAC 17B supplies a referencevoltage for AD conversion to the ADC 17A and, broadly speaking, isincluded in a part of the ADC 17. Although not illustrated in FIG. 8 ,the selector 16 is also arranged on the second substrate 120.

The second substrate 120 also has wiring 122 in contact with andelectrically connected to the TSVs in the TSV array 102 passing throughthe first substrate 100 (hereinafter simply referred to as TSV array102), and a pad array 123 in which a plurality of pads electricallyconnected to the pads in the pad array 103 of the first substrate 100are arranged linearly.

For the connection between the TSV array 102 and the wiring 122, forexample, the following technology can be employed: twin TSV technologyin which two TSVs, namely, a TSV provided in the first substrate 100 anda TSV provided from the first substrate 100 to the second substrate 120are connected with the chip facing out, or shared TSV technology inwhich a shared TSV provided from the first substrate 100 to the secondsubstrate 120 provides connection. However, the present invention is notlimited thereto, and a variety of connection modes can be employed.Examples include Cu—Cu bonding in which copper (Cu) exposed on the jointsurface of the first substrate 100 and Cu exposed on the joint surfaceof the second substrate 120 are joined.

The connection mode between the pads in the pad array 103 on the firstsubstrate 100 and the pads in the pad array 123 of the second substrate120 may be, for example, wire bonding. However, the present invention isnot limited thereto, and connection modes such as through holes andcastellation may be employed.

In a layout example of the second substrate 120, for example, the ADC17A, the signal processor 13, and the DSP 14 are arranged in order fromthe upstream side along the flow of a signal read out from the pixelarray 101, where the upstream side is the vicinity of the wiring 122connected to the TSV array 102. That is, the ADC 17A to which a pixelsignal read out from the pixel array 101 is initially input is arrangedin the vicinity of the wiring 122 on the most upstream side, next thesignal processor 13 is arranged, and the DSP 14 is arranged in a regionfarthest from the wiring 122. Such a layout in which the ADC 17 to theDSP 14 are arranged from the upstream side along the flow of a signalcan shorten the wiring connecting the parts. This layout leads toreduction in signal delay, reduction in signal propagation loss,improvement of the S/N ratio, and lower power consumption.

The controller 12 is arranged, for example, in the vicinity of thewiring 122 on the upstream side. In FIG. 8 , the controller 12 isarranged between the ADC 17A and the signal processor 13. Such a layoutleads to reduction in signal delay, reduction in signal propagationloss, improvement of the S/N ratio, and lower power consumption when thecontroller 12 controls the pixel array 101. Advantageously, the signalpin and the power supply pin for analog circuits can be collectivelyarranged in the vicinity of the analog circuits (for example, in thelower side of FIG. 8 ), the remaining signal pin and power supply pinfor digital circuits can be collectively arranged in the vicinity of thedigital circuits (for example, in the upper side of FIG. 8 ), or thepower supply pin for analog circuits and the power supply pin fordigital circuits can be sufficiently spaced apart from each other.

In the layout illustrated in FIG. 8 , the DSP 14 is arranged on the sideopposite to the ADC 17A on the most downstream side. With such a layout,in other words, the DSP 14 can be arranged in a region not overlappingwith the pixel array 101 in the stacking direction of the firstsubstrate 100 and the second substrate 120 (hereinafter simply referredto as top-bottom direction).

In this way, in the configuration in which the pixel array 101 and theDSP 14 are not superimposed in the top-bottom direction, intrusion ofnoise produced due to signal processing by the DSP 14 into the pixelarray 101 can be reduced. As a result, even when the DSP 14 operates asa processing unit that performs computation based on a pre-trainedmodel, intrusion of noise resulting from signal processing by the DSP 14into the pixel array 101 can be reduced, and consequently, an image withless deterioration in quality can be acquired.

The DSP 14 and the signal processor 13 are connected by an interconnect14 a configured with a part of the DSP 14 or a signal line. The selector16 is arranged, for example, in the vicinity of the DSP 14. When theinterconnect 14 a is a part of the DSP 14, the DSP 14 may partiallyoverlap with the pixel array 101 in the top-bottom direction. However,even in such a case, compared with when the whole of the DSP 14 issuperimposed on the pixel array 101 in the top-bottom direction,intrusion of noise into the pixel array 101 can be reduced.

Memories 15A and 15B are arranged, for example, so as to surround theDSP 14 from three directions. In such an arrangement of the memories 15Aand 15B surrounding the DSP 14, the distance of wiring between eachmemory element in the memory 15 and the DSP 14 can be averaged while thedistance can be reduced as a whole. Consequently, signal delay, signalpropagation loss, and power consumption can be reduced when the DSP 14accesses the memory 15.

The pad array 123 is arranged, for example, at a position on the secondsubstrate 120 corresponding to the pad array 103 of the first substrate100 in the top-bottom direction. Here, among the pads included in thepad array 123, a pad positioned in the vicinity of the ADC 17A is usedfor propagation of power supply voltage for analog circuits (mainly theADC 17A) or an analog signal. On the other hand, a pad positioned in thevicinity of the controller 12, the signal processor 13, the DSP 14, orthe memories 15A and 15B is used for propagation of power supply voltagefor digital circuits (mainly, the controller 12, the signal processor13, the DSP 14, the memories 15A and 15B) and a digital signal. Such apad layout can reduce the distance of wiring connecting the pads to theparts. This layout leads to reduction in signal delay, reduction inpropagation loss of signals and power supply voltage, improvement of theS/N ratio, and lower power consumption.

6. Other Embodiments

The process according to the foregoing embodiments may be carried out invarious manners in addition to the foregoing embodiments.

The numerical values such as frame rate and predetermined time describedin the foregoing embodiments are illustrative only and can be changed asdesired. The contents of the DSP process described in the embodimentsare also illustrative only, and a variety of processes using a learningmodel can be employed. The DSP process may be performed when the enablesignal is inactive (OFF), and the image readout process may be performedwhen the enable signal is active (ON).

In the foregoing embodiments, a learning model that has learned via aDNN is described by way of example. However, in addition to DNNs,various neural networks such as recurrent neural networks (RNNs) andconvolutional neural networks (CNNs) can be used. The present inventionis not limited to a learning model using a DNN and the like, andlearning models that have learned through other various machine learningsuch as decision trees and support vector machines can be used.

The information including process procedures, control procedures,specific names, a variety of data and parameters shown in thedescription and the drawings can be changed as desired unless otherwisespecified. The specific examples, distributions, and numerical valuesdescribed in the examples are illustrative only and can be changed asdesired.

The constituent elements in the devices illustrated in the drawings arefunctional and conceptual and are not necessarily physically configuredas depicted in the drawings. That is, a specific manner of distributionand integration of the devices is not limited to the one illustrated inthe drawings, and the whole or a part thereof may be functionally orphysically distributed or integrated in any units, depending on avariety of loads and use situations. For example, the controller 12 andthe signal processor 13 illustrated in FIG. 1 may be integrated.

7. Application to Movable Body

The technique according to the present disclosure (the presenttechnique) is applicable to a variety of products. For example, thetechnique according to the present disclosure may be implemented as adevice mounted on any type of movable bodies, such as automobiles,electric vehicles, hybrid electric vehicles, motorcycles, bicycles,personal mobility devices, airplanes, drones, vessels and ships, androbots.

FIG. 10 is a block diagram illustrating an example of the overallconfiguration of a vehicle control system that is an example of amovable body control system to which the technique according to thepresent disclosure is applicable.

A vehicle control system 12000 includes a plurality of electroniccontrol units connected through a communication network 12001. In theexample illustrated in FIG. 10 , the vehicle control system 12000includes a drive control unit 12010, a body control unit 12020, avehicle exterior information detection unit 12030, a vehicle interiorinformation detection unit 12040, and a central control unit 12050. As afunctional configuration of the central control unit 12050, amicrocomputer 12051, a sound image output module 12052, and anin-vehicle network I/F (interface) 12053 are illustrated.

The drive control unit 12010 controls operation of devices related to adrive system of a vehicle in accordance with a variety of computerprograms. For example, the drive control unit 12010 functions as acontrol device for a drive force generating device for generating driveforce of the vehicle, such as an internal combustion engine or a drivemotor, a drive force transmission mechanism for transmitting drive forceto the wheels, a steering mechanism for adjusting the steering angle ofthe vehicle, and a braking device for generating braking force of thevehicle.

The body control unit 12020 controls operation of a variety of devicesinstalled in the vehicle body in accordance with a variety of computerprograms. For example, the body control unit 12020 functions as acontrol device for a keyless entry system, a smart key system, a powerwindow device, or a variety of lamps such as head lamps, rear lamps,brake lamps, turn signals, and fog lamps. In this case, the body controlunit 12020 may receive radio waves transmitted from a portable devicealternative to a key or signals from a variety of switches. The bodycontrol unit 12020 accepts input of the radio waves or signals andcontrols a door lock device, a power window device, a lamp, and the likeof the vehicle.

The vehicle exterior information detection unit 12030 detectsinformation on the outside of the vehicle equipped with the vehiclecontrol system 12000. For example, an imager 12031 is connected to thevehicle exterior information detection unit 12030. The vehicle exteriorinformation detection unit 12030 allows the imager 12031 to capture animage of the outside of the vehicle and receives the captured image. Thevehicle exterior information detection unit 12030 may perform an objectdetection process or a distance detection process for persons, vehicles,obstacles, signs, or characters on roads, based on the received image.

The imager 12031 is an optical sensor that receives light and outputs anelectrical signal corresponding to the quantity of received light of thelight. The imager 12031 may output an electrical signal as an image oroutput as information on a measured distance. Light received by theimager 12031 may be visible light or invisible light such as infraredrays.

The vehicle interior information detection unit 12040 detectsinformation on the inside of the vehicle. The vehicle interiorinformation detection unit 12040 is connected to, for example, a driverstate detector 12041 that detects a state of the driver. The driverstate detector 12041 includes, for example, a camera for taking an imageof the driver, and the vehicle interior information detection unit 12040may calculate the degree of fatigue or the degree of concentration ofthe driver or may determine whether the driver falls asleep, based ondetection information input from the driver state detector 12041.

The microcomputer 12051 can compute a control target value for the driveforce generating device, the steering mechanism, or the braking device,based on information on the inside and outside of the vehicle acquiredby the vehicle exterior information detection unit 12030 or the vehicleinterior information detection unit 12040, and output a control commandto the drive control unit 12010. For example, the microcomputer 12051can perform coordination control for the purpose of functionimplementation of advanced driver assistance systems (ADAS), includingcollision avoidance or shock mitigation of the vehicle, car-followingdrive based on the distance between vehicles, vehicle speed-keepingdrive, vehicle collision warning, and lane departure warning.

The microcomputer 12051 can perform coordination control for the purposeof, for example, autonomous driving, in which the drive force generatingdevice, the steering mechanism, or the braking device is controlledbased on information on the surroundings of the vehicle acquired by thevehicle exterior information detection unit 12030 or the vehicleinterior information detection unit 12040 to enable autonomous drivingwithout depending on the operation by the driver.

The microcomputer 12051 can output a control command to the body controlunit 12020, based on information on the outside of the vehicle acquiredby the vehicle exterior information detection unit 12030. For example,the microcomputer 12051 can perform coordination control for theantidazzle purpose, for example, by controlling the head lamps inaccordance with the position of a vehicle ahead or an oncoming vehicledetected by the vehicle exterior information detection unit 12030 toswitch high beams to low beams.

The sound image output module 12052 transmits an output signal of atleast one of sound and image to an output device capable of visually oraurally giving information to a passenger in the vehicle or the outsideof the vehicle. In the example in FIG. 10 , an audio speaker 12061, adisplay 12062, and an instrument panel 12063 are illustrated as theoutput device. The display 12062 may include, for example, at least oneof an on-board display and a head-up display.

FIG. 11 is a diagram illustrating an example of the installationposition of the imager 12031.

In FIG. 11 , imagers 12101, 12102, 12103, 12104, and 12105 are providedas the imager 12031.

The imagers 12101, 12102, 12103, 12104, and 12105 are provided, forexample, at positions such as front nose, side mirrors, rear bumper,back door of a vehicle 12100, and an upper portion of the front glassinside the vehicle. The imager 12101 provided at the front nose and theimager 12105 provided at the upper portion of the front glass inside thevehicle mainly acquire an image in front of the vehicle 12100. Theimagers 12102 and 12103 provided at the side mirrors mainly acquireimages on the sides of the vehicle 12100. The imager 12104 provided atthe rear bumper or the back door mainly acquires an image behind thevehicle 12100. The imager 12105 provided at the upper portion of thefront glass in the vehicle interior is mainly used for detecting avehicle ahead, pedestrians, obstacle, traffic signs, road signs, trafficlanes, and the like.

FIG. 11 illustrates an example of the imaging ranges of the imagers12101 and 12104. An imaging range 12111 indicates an imaging range ofthe imager 12101 provided at the front nose, imaging ranges 12112 and12113 indicate the imaging ranges of the imagers 12102 and 12103provided at the side mirrors, and an imaging range 12114 indicates theimaging range of the imager 12104 provided at the rear bumper or theback door. For example, a bird's eye view of the vehicle 12100 viewedfrom above can be obtained by superimposing image data captured by theimagers 12101 and 12104.

At least one of the imagers 12101 and 12104 may have a function ofacquiring distance information. For example, at least one of the imagers12101 and 12104 may be a stereo camera including a plurality of imagesensors or may be an image sensor having a pixel for phase differencedetection.

For example, the microcomputer 12051 can obtain the distance to athree-dimensional object within the imaging range 12111 or 12114 and atemporal change of this distance (relative speed to the vehicle 12100),based on distance information obtained from the imager 12101 or 12104,to specifically extract a three-dimensional object closest to thevehicle 12100 on the path of travel and traveling at a predeterminedspeed (for example, 0 km/h or more) in substantially the same directionas the vehicle 12100, as a vehicle ahead. In addition, the microcomputer12051 can preset a distance between vehicles to be kept in front of avehicle ahead and perform, for example, automatic braking control(including car-following stop control) and automatic speed-up control(including car-following startup control). In this way, coordinationcontrol can be performed, for example, for the purpose of autonomousdriving in which the vehicle runs autonomously without depending on theoperation by the driver.

For example, the microcomputer 12051 can classify three-dimensionalobject data on a three-dimensional object into two-wheel vehicle,standard-sized vehicle, heavy vehicle, pedestrian, utility pole, or anyother three-dimensional object, based on the distance informationobtained from the imager 12101 or 12104, and can use the extracted datafor automatic avoidance of obstacles. For example, the microcomputer12051 identifies an obstacle in the surroundings of the vehicle 12100 asan obstacle visible to the driver of the vehicle 12100 or as an obstaclehardly visible. The microcomputer 12051 then determines a collision riskindicating the degree of risk of collision with each obstacle and, whenthe collision risk is equal to or higher than a setting value and thereis a possibility of collision, outputs an alarm to the driver throughthe audio speaker 12061 or the display 12062, or performs forceddeceleration or avoidance steering through the drive control unit 12010,thereby implementing drive assistance for collision avoidance.

At least one of the imagers 12101 and 12104 may be an infrared camerathat detects infrared rays. For example, the microcomputer 12051 canrecognize a pedestrian by determining whether a pedestrian exists in thecaptured image by the imager 12101 or 12104. Such recognition ofpedestrians is performed, for example, through the procedure ofextracting feature points in the captured image by the imager 12101 or12104 serving as an infrared camera and the procedure of performingpattern matching with a series of feature points indicating the outlineof an object to determine whether the object is a pedestrian. When themicrocomputer 12051 determines that a pedestrian exists in the capturedimage by the imager 12101 or 12104 and recognizes a pedestrian, thesound image output module 12052 controls the display 12062 such that arectangular outline for highlighting the recognized pedestrian issuperimposed. The sound image output module 12052 may control thedisplay 12062 such that an icon indicating a pedestrian appears at adesired position.

An example of the vehicle control system to which the techniqueaccording to the present disclosure is applicable has been describedabove. The technique according to the present disclosure is applicableto the imager 12031 and the like in the configuration described above.When the technique according to the present disclosure is applied to theimager 12031 and the like, miniaturization of the imager 12031 and thelike can be achieved, thereby facilitating design of the interior andthe exterior of the vehicle 12100. When the technique according to thepresent disclosure is applied to the imager 12031 and the like, a clearimage with reduced noise can be acquired to provide a driver with a morevisible image. Consequently, the driver's fatigue can be alleviated.

8. Application to Endoscopic Surgery System

The technique according to the present disclosure (the presenttechnique) is applicable to a variety of products. For example, thetechnique according to the present disclosure may be applied to anendoscopic surgery system.

FIG. 12 is a diagram illustrating an example of the overallconfiguration of an endoscopic surgery system to which the techniqueaccording to the present disclosure (the present technique) isapplicable.

FIG. 12 illustrates a situation in which an operator (doctor) 11131 usesan endoscopic surgery system 11000 to perform an operation on a patient11132 on a patient bed 11133. As illustrated in the drawing, theendoscopic surgery system 11000 includes an endoscope 11100, othersurgical instruments 11110 such as an insufflation tube 11111 and anenergy treatment tool 11112, a support arm device 11120 supporting theendoscope 11100, and a cart 11200 carrying a variety of devices forendoscopic surgery.

The endoscope 11100 includes a barrel 11101 having a region of apredetermined length from its tip end to be inserted into the bodycavity of the patient 11132, and a camera head 11102 connected to thebase end of the barrel 11101. In the example illustrated in the drawing,the endoscope 11100 is a rigid borescope having a rigid barrel 11101.However, the endoscope 11100 may be configured as a soft borescopehaving a soft barrel.

The tip end of the barrel 11101 has an opening having an objective lensfitted therein. A light source device 11203 is connected to theendoscope 11100. Light generated by the light source device 11203 ispropagated to the tip end of the barrel through a light guide extendinginside the barrel 11101 and irradiates an observation target in the bodycavity of the patient 11132 through the objective lens. The endoscope11100 may be a forward-viewing endoscope or may be a forward-obliqueviewing endoscope or a side-viewing endoscope.

An optical system and an image sensor are provided inside the camerahead 11102. Reflected light (observation light) from an observationtarget is collected by the optical system onto the image sensor. Theobservation light is converted to electricity by the image sensor togenerate an electrical signal corresponding to the observation light,that is, an image signal corresponding to an observation image. Theimage signal is transmitted as RAW data to a camera control unit (CCU)11201.

The CCU 11201 is configured with a central processing unit (CPU), agraphics processing unit (GPU), or the like to centrally control theoperation of the endoscope 11100 and a display device 11202. The CCU11201 receives an image signal from the camera head 11102 and performs avariety of image processing on the image signal, for example, adevelopment process (demosaicing) for displaying an image based on theimage signal.

The display device 11202 displays an image based on the image signalsubjected to image processing by the CCU 11201, under the control of theCCU 11201.

The light source device 11203 is configured with a light source such asa light emitting diode (LED) and supplies the endoscope 11100 withradiation light in imaging a surgery site.

An input device 11204 is an input interface with the endoscopic surgerysystem 11000. The user can input a variety of information andinstructions to the endoscopic surgery system 11000 through the inputdevice 11204. For example, the user inputs an instruction to change theimaging conditions by the endoscope 11100 (the kind of radiation light,magnification, focal length, etc.).

A treatment tool control device 11205 controls actuation of the energytreatment tool 11112 for cauterization of tissues, incision, or sealingof blood vessels. An insufflator 11206 feeds gas into the body cavitythrough the insufflation tube 11111 to insufflate the body cavity of thepatient 11132 in order to ensure the field of view with the endoscope11100 and ensure a working space for the operator. A recorder 11207 is adevice capable of recording a variety of information on surgery. Aprinter 11208 is a device capable of printing a variety of informationon surgery in a variety of forms such as text, image, or graph.

The light source device 11203 that supplies the endoscope 11100 withradiation light in imaging a surgery site can be configured with, forexample, a white light source such as an LED, a laser light source, or acombination thereof. When a white light source is configured with acombination of RGB laser light sources, the output power and the outputtiming of each color (each wavelength) can be controlled accurately,and, therefore, the white balance of the captured image can be adjustedin the light source device 11203. In this case, an observation target isirradiated time-divisionally with laser light from each of the RGB laserlight sources, and actuation of the image sensor in the camera head11102 is controlled in synchronization with the radiation timing,whereby an image corresponding to each of R, G, and B can be capturedtime-divisionally. According to this method, a color image can beobtained even without color filters in the image sensor.

The actuation of the light source device 11203 may be controlled suchthat the intensity of output light is changed every certain time. Insynchronization with the timing of changing the intensity of light, theactuation of the image sensor in the camera head 11102 is controlled toacquire images time-divisionally, and the images are combined togenerate an image with a high dynamic range free from blocked-up shadowsand blown out highlights.

The light source device 11203 may be configured to supply light in apredetermined wavelength band corresponding to specific lightobservation. In specific light observation, for example, narrow bandimaging is performed, which uses the wavelength dependency of lightabsorption in body tissues and applies light in a narrow band, comparedwith radiation light (that is, white light) in normal observation, tocapture an image of predetermined tissues such as blood vessels in theoutermost surface of mucosa. Alternatively, in specific lightobservation, fluorescence observation may be performed in which an imageis acquired by fluorescence generated by radiation of excitation light.In fluorescence observation, for example, excitation light is applied tobody tissues and fluorescence from the body tissues is observed(autofluorescence imaging), or a reagent such as indocyanine green (ICG)is locally injected to body tissues and excitation light correspondingto the fluorescence wavelength of the reagent is applied to the bodytissues to obtain a fluorescence image. The light source device 11203may be configured to supply narrow-band light and/or excitation lightcorresponding to such specific light observation.

FIG. 13 is a block diagram illustrating an example of the functionalconfiguration of the camera head 11102 and the CCU 11201 illustrated inFIG. 12 .

The camera head 11102 includes a lens unit 11401, an imager 11402, adriver 11403, a communication module 11404, and a camera head controller11405. The CCU 11201 includes a communication module 11411, an imageprocessor 11412, and a controller 11413. The camera head 11102 and theCCU 11201 are connected to communicate with each other through atransmission cable 11400.

The lens unit 11401 is an optical system provided at a connectionportion to the barrel 11101. Observation light taken in from the tip endof the barrel 11101 is propagated to the camera head 11102 and entersthe lens unit 11401. The lens unit 11401 is configured with acombination of a plurality of lenses including a zoom lens and a focuslens.

The imager 11402 may be configured with one image sensor (called singlesensor-type) or a plurality of image sensors (called multi sensor-type).When the imager 11402 is a multi-sensor construction, for example, imagesignals corresponding to R, G, and B may be generated by image sensorsand combined to produce a color image. Alternatively, the imager 11402may have a pair of image sensors for acquiring image signals for righteye and for left eye corresponding to three-dimensional (3D) display.The 3D display enables the operator 11131 to more accurately grasp thedepth of living tissues in a surgery site. When the imager 11402 is amulti-sensor construction, several lines of lens units 11401 may beprovided corresponding to the image sensors.

The imager 11402 is not necessarily provided in the camera head 11102.For example, the imager 11402 may be provided immediately behind theobjective lens inside the barrel 11101.

The driver 11403 is configured with an actuator and moves the zoom lensand the focus lens of the lens unit 11401 by a predetermined distancealong the optical axis under the control of the camera head controller11405. The magnification and the focal point of a captured image by theimager 11402 thus can be adjusted as appropriate.

The communication module 11404 is configured with a communication devicefor transmitting/receiving a variety of information to/from the CCU11201. The communication module 11404 transmits an image signal obtainedfrom the imager 11402 as RAW data to the CCU 11201 through thetransmission cable 11400.

The communication module 11404 receives a control signal for controllingactuation of the camera head 11102 from the CCU 11201 and supplies thereceived signal to the camera head controller 11405. The control signalincludes, for example, information on imaging conditions, such asinformation specifying a frame rate of the captured images, informationspecifying an exposure value in imaging, and/or information specifying amagnification and a focal point of the captured image.

The image conditions such as frame rate, exposure value, magnification,and focal point may be specified as appropriate by the user or may beautomatically set by the controller 11413 of the CCU 11201 based on theacquired image signal. In the latter case, the endoscope 11100 isequipped with an auto exposure (AE) function, an auto focus (AF)function, and an auto white balance (AWB) function.

The camera head controller 11405 controls actuation of the camera head11102, based on a control signal received from the CCU 11201 through thecommunication module 11404.

The communication module 11411 is configured with a communication devicefor transmitting/receiving a variety of information to/from the camerahead 11102. The communication module 11411 receives an image signaltransmitted from the camera head 11102 through the transmission cable11400.

The communication module 11411 transmits a control signal forcontrolling actuation of the camera head 11102 to the camera head 11102.The image signal and the control signal can be transmitted viaelectrical communication or optical communication.

The image processor 11412 performs a variety of image processing on theimage signal that is RAW data transmitted from the camera head 11102.

The controller 11413 performs a variety of control on imaging of asurgery site and the like by the endoscope 11100 and display of acaptured image obtained by imaging of a surgery site and the like. Forexample, the controller 11413 generates a control signal for controllingactuation of the camera head 11102.

The controller 11413 displays a captured image visualizing a surgerysite and the like on the display device 11202, based on the image signalsubjected to image processing by the image processor 11412. In doing so,the controller 11413 may recognize a variety of objects in the capturedimage using a variety of image recognition techniques. For example, thecontroller 11413 detects the shape of edge, color, and the like of anobject included in the captured image to recognize a surgical instrumentsuch as forceps, a specific living body site, bleeding, and mist in useof the energy treatment tool 11112. When displaying the captured imageon the display device 11202, the controller 11413 may use therecognition result to superimpose a variety of surgery assistinginformation on the image of the surgery site. The surgery assistinginformation superimposed and presented to the operator 11131 canalleviate burden on the operator 11131 or ensure the operator 11131 toproceed surgery.

The transmission cable 11400 connecting the camera head 11102 and theCCU 11201 is an electrical signal cable corresponding to communicationof electrical signals, an optical fiber corresponding to opticalcommunication, or a composite cable thereof.

In the example illustrated in the drawing, the transmission cable 11400is used for wired communication. However, communication between thecamera head 11102 and the CCU 11201 may be wireless.

An example of the endoscopic surgery system to which the techniqueaccording to the present disclosure is applicable has been describedabove. The technique according to the present disclosure is applicableto, for example, the imager 11402 and the like in the camera head 11102.When the technique according to the present disclosure is applied to thecamera head 11102, the camera head 11102 and the like can beminiaturized, resulting in the compact endoscopic surgery system 11000.When the technique according to the present disclosure is applied to thecamera head 11102 and the like, a clear image with reduced noise can beacquired to provide the operator with a more visible image.Consequently, the operator's fatigue can be alleviated.

Although the endoscopic surgery system has been described here by way ofexample, the technique according to the present disclosure may beapplied to, for example, a microscopic surgery system.

The foregoing embodiments and modifications can be combined asappropriate as far as the process contents are consistent.

The effects described in the present description are illustrative onlyand not limitative, and any other effects may be contemplated.

Note that the present technology can also have the followingconfigurations.

(1)

A solid-state imaging device comprising:

an imager configured to acquire image data;

a processing unit configured to perform a process based on a neuralnetwork calculation model for data based on the image data acquired fromthe imager; and

a control unit configured to switch between a first process mode ofperforming a first process at a first frame rate and, based on a resultof the first process, a second process mode of performing a secondprocess at a second frame rate.

(2)

The solid-state imaging device according to (1), wherein the processingunit performs a computation process of detecting a certain detectiontarget, for data based on the image data.

(3)

The solid-state imaging device according to (2), wherein the computationprocess is a process using a pre-trained learning model.

(4)

The solid-state imaging device according to any one of (1) to (3),wherein the second process is a process that does not perform a processbased on a neural network calculation model for data based on the imagedata acquired from the imager.

(5)

The solid-state imaging device according to (4), wherein the secondprocess is a process of reading out the image data at the second framerate that is higher than the first frame rate, without performing aprocess based on a neural network calculation model for data based onthe image data acquired from the imager, when a certain detection targetis detected in the first process mode.

(6)

The solid-state imaging device according to any one of (1) to (3),wherein the second process is a process in which a process of readingout image data from the imager is temporally overlapped with a processof performing a process based on a neural network calculation model fordata based on the image data acquired from the imager.

(7)

The solid-state imaging device according to (6), wherein the secondprocess is a process of performing a process based on a neural networkcalculation model for data based on the image data acquired from theimager, at the second frame rate that is the same frame rate as thefirst frame rate, when a certain detection target is detected in thefirst process mode.

(8)

The solid-state imaging device according to (7), wherein the secondprocess is a process of performing a process based on a neural networkcalculation model for data based on the image data acquired from theimager, at the second frame rate that is higher than the first framerate, when a certain detection target is detected in the first processmode.

(9)

The solid-state imaging device according to any one of (2) to (8),wherein the control unit starts the process based on the neural networkcalculation model for data based on the image data by the computationprocess, after readout of the image data from the imager is completed ina state in which the first process mode is selected.

(10)

The solid-state imaging device according to any one of (2) to (9),wherein the control unit starts readout of the image data from theimager, after the computation process is completed in a state in whichthe first process mode is selected.

(11)

An electronic device comprising:

a solid-state imaging device including an imager configured to acquireimage data, a processing unit configured to perform a process based on aneural network calculation model for data based on the image dataacquired from the imager, and a control unit configured to switchbetween a first process mode of performing a first process at a firstframe rate and, based on a result of the first process, a second processmode of performing a second process at a second frame rate; and

a control device configured to perform a process by an application forthe image data output from the solid-state imaging device.

REFERENCE SIGNS LIST

1 imaging device

10 image sensor

11 imager

12 controller

13 signal processor

14 DSP (processing unit)

15 memory

16 selector

20 application processor

30 cloud server

1. A solid-state imaging device comprising: an imager configured toacquire image data; a processing unit configured to perform a processbased on a neural network calculation model for data based on the imagedata acquired from the imager; and a control unit configured to switchbetween a first process mode of performing a first process at a firstframe rate and, based on a result of the first process, a second processmode of performing a second process at a second frame rate.
 2. Thesolid-state imaging device according to claim 1, wherein the processingunit performs a computation process of detecting a certain detectiontarget, for data based on the image data.
 3. The solid-state imagingdevice according to claim 2, wherein the computation process is aprocess using a pre-trained learning model.
 4. The solid-state imagingdevice according to claim 1, wherein the second process is a processthat does not perform a process based on a neural network calculationmodel for data based on the image data acquired from the imager.
 5. Thesolid-state imaging device according to claim 4, wherein the secondprocess is a process of reading out the image data at the second framerate that is higher than the first frame rate, without performing aprocess based on a neural network calculation model for data based onthe image data acquired from the imager, when a certain detection targetis detected in the first process mode.
 6. The solid-state imaging deviceaccording to claim 1, wherein the second process is a process in which aprocess of reading out image data from the imager is temporallyoverlapped with a process of performing a process based on a neuralnetwork calculation model for data based on the image data acquired fromthe imager.
 7. The solid-state imaging device according to claim 6,wherein the second process is a process of performing a process based ona neural network calculation model for data based on the image dataacquired from the imager, at the second frame rate that is the sameframe rate as the first frame rate, when a certain detection target isdetected in the first process mode.
 8. The solid-state imaging deviceaccording to claim 7, wherein the second process is a process ofperforming a process based on a neural network calculation model fordata based on the image data acquired from the imager, at the secondframe rate that is higher than the first frame rate, when a certaindetection target is detected in the first process mode.
 9. Thesolid-state imaging device according to claim 2, wherein the controlunit starts the process based on the neural network calculation modelfor data based on the image data by the computation process, afterreadout of the image data from the imager is completed in a state inwhich the first process mode is selected.
 10. The solid-state imagingdevice according to claim 2, wherein the control unit starts readout ofthe image data from the imager, after the computation process iscompleted in a state in which the first process mode is selected.
 11. Anelectronic device comprising: a solid-state imaging device including animager configured to acquire image data, a processing unit configured toperform a process based on a neural network calculation model for databased on the image data acquired from the imager, and a control unitconfigured to switch between a first process mode of performing a firstprocess at a first frame rate and, based on a result of the firstprocess, a second process mode of performing a second process at asecond frame rate; and a control device configured to perform a processby an application for the image data output from the solid-state imagingdevice.